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1.  Background 


This  report  is  presented  in  the  form  of  an  executive  summary  attached  to  a  more  detailed 
discussion  paper  covering  both  the  theoretical  and  applied  findings  of  research  undertaken  at 
the  University  of  Queensland  with  support  of  AOARD  award  number  FA2386-10-1-4011. 

The  project  commenced  with  the  objective  of  developing  a  performance  assessment  tool  able 
to  measure  instantaneous  mood  state  (affect)  of  individuals  during  simulation  training.  The 
project's  primary  objective  was  to  investigate  affect  as  a  measure  of  non-technical  skills  (such 
as  decision-making)  in  the  expectation  that  it  would  offer  a  more  direct  measure  than  current 
alternatives.  During  the  past  year  the  project  has  developed  a  further  emphasis  with  affective 
skills  themselves  becoming  a  direct  variable  of  interest.  The  objective  affect  measurement  tool 
(OAMT)  will  allow  affective  skill  levels  of  an  individual  during  task  performance  to  be  evaluated. 

The  original  project  aims  were  to:  Develop  a  method  and  associated  theoretical  framework  for 
recognizing  user  affective  states  during  simulation-based  training.  To  validate  the  face-based 
affect  recognition  prototype  method  through  application  in  experimental  simulated  training 
scenarios,  and  systematically  explore  feature  extraction  as  an  objective  measurement  approach 
to  evaluate  the  impact  of  learner  affective  states  on  the  effectiveness  of  complex  cognitive 
training  via  simulation. 

It  had  three  original  broad  objectives; 

a)  To  examine  the  relationships  between  affective  intensity  and  cognitive  learning  and  the 
impact  they  have  on  facial  features,  perceptual  processes,  physiological  response  and 
electromyographic  activity. 

b)  To  identify  a  reliable  set  of  objectively  measurable  features  and  responses  that  highly 
correlates  to  target  emotions  and  cognitive  functions. 

c)  To  build  theoretical  tools  that  support  cognitive  training  in  simulation  initially  from  existing 
theoretical  insights  with  novel  theoretical  insights  added  as  new  links  are  discovered  in  the 
network  of  associations  surrounding  the  construct  of  affect. 

The  project  plan  was  designed  to  allow  for  two  separate  pilot  tests  of  the  technical  equipment: 

Pilot  One  was  undertaken  at  the  experimental  simulation  laboratory  located  on  the  St  Lucia 
campus  of  the  University  of  Queensland  using  subjects  drawn  from  the  University  population. 
The  sole  purpose  of  Pilot  One  was  to  achieve  proof  of  concept  and  trialing/interfacing  of  the 
measurement  equipment.  The  attached  discussion  paper  outlines  in  detail  the  technical  set-up 
work  undertaken.  While  the  simulator  task  was  not  as  stressful  as  tasks  undertaken  during 
simulator  training  in  a  real  world  setting,  measures  obtained  from  the  physiological  monitoring 
equipment  revealed  that  anxiety  levels  were  reflected  in  a  range  of  physiological  data. 


Pilot  Two  was  to  test  the  equipment  set-up  in  real  world  simulator  training  setting  in  which 
anxiety  and  stress  levels  would  be  expected  to  be  much  higher.  Successful  negotiations  were 
undertaken  with  a  flight  training  program.  The  plan  was  to  monitor  trainee  pilots  drawn  from  a 
university  Bachelor  of  Aviation  Degree  while  they  undertook  testing  in  a  flight  simulator.  The 
experiment  is  set  up  and  ready  to  run  but,  unfortunately,  while  AFOSR  IRB  approval  was  sought 
in  May  2010,  approval  has,  at  the  time  of  writing,  not  been  granted.  Thus  we  are  unable  to 
report  the  results  of  Pilot  Two  in  this  report. 

2.  Affective  Skills  Theory 

During  the  twelve  months  of  the  project  theoretical  advances  were  made  resulting  in  the 
Affective  Skills  Theory  being  developed  to  explain  the  phenomenon  whereby  professionals, 
repeatedly  exposed  to  high  stress  conditions  in  a  simulator,  can  gain  the  affective  skills  to 
control  and  use  their  emotions  to  achieve  high  performance.  The  existence  of  affective  skills 
was  also  supported  by  scientific  findings  in  other  fields  of  enquiry  such  as  elite  sport  where 
'affective  intelligence'  is  terminology  used  to  describe  the  ability  to  remain  in  control  of 
emotions.  The  field  actively  promotes  the  channeling  of  emotions  to  facilitate  optimal 
performance. 

In  Affective  Skill  Theory  (AST),  affect  is  therefore  conceptualized  as  a  skill  that  can  be  trained 
prior  to  an  operation.  As  a  skill,  affect  can  also  be  used  as  a  key  variable  in  evaluating  operation 
success  based  on  the  ability  to  assess  the  match  between  the  affective  skills  of  personnel  with 
overall  mission  affective  requirements.  The  theory  has  the  potential  to  be  highly  valuable  in 
supporting  operational  planning  through  identifying  situations  in  the  field  which  will  impose 
high  affective  task  demands  on  personnel  and  support  assessment  of  the  optimal  affective  level 
exhibited  by  experienced  personnel  able  to  successfully  complete  the  operation. 

3.  Affective  Skills  Theory  in  Workload  Prediction  and  Operational  Planning 

Real-time  monitoring  of  individuals  for  affective  skill  capacity  via  an  approach  based  on 
sampling  multiple  neurophysiologic  signals,  and  integrating  those  signals  with  a  performance 
prediction  model  able  to  estimate  the  affective  component  of  workload,  potentially  provides  a 
new  method  of  supporting  command  and  control  personnel. 

A  number  of  workload  prediction  models  have  been  developed  and  are  currently  in  use  within 
defense.  To  date,  predictive  workload  measures  analyze  a  task/workload  estimating  cognitive, 
psychomotor,  auditory,  kinesthetic,  visual  and  sensory  components  of  workload  for  each  task. 
To  our  knowledge,  no  current  prediction  models  analyze  the  affective  cost  of  an  operation. 

The  Affective  Skill  Theory  developed  in  the  first  phase  of  the  current  AOARD  project  supports 
the  development  of  a  predictive  workload  measure  able  to  estimate  affective  workload.  By 
conceptualizing  affect  as  a  professional  skill  that  can  be  trained  prior  to  an  operation,  affective 
skills  are  recognized  as  a  key  variable  when  assessing  both  performance  and  workload.  Being 
able  to  determine  the  overall  affective  costs  of  a  planned  mission  has  significant  potential  to 


assist  command  and  control  in  the  area  of  operational  planning.  For  example,  a  model  of  task 
workload  which  accounts  for  affective  output  (the  affective  costs  of  a  deployment)  will  support 
the  identification  of  situations  in  the  field  which  will  impose  high  affective  task  demands  on 
personnel.  When  an  operation  has  been  identified  as  high  in  affective  demand,  the  optimal 
affective  level  of  expertise  required  of  personnel  to  successfully  complete  the  operation  can  be 
estimated.  Ultimately,  such  a  theoretical  model  that  facilitates  the  mapping  of  affective 
requirements  and  cost  with  task  demands  could  assist  in  predictions  of  overall  mission  success 
and  in  the  planning  of  additional  training  requirements. 

This  new  direction  in  operational  planning  support  relies  on  the  development  of  the  objective 
affect  measurement  tool  (OAMT)  occurring  concurrently  with  the  development  of  a  predictive 
computational  affect  workload  model  (CAWM).  Affective  state  is  a  key  variable  within 
computational  prediction  models  currently  under  development  at  UQ.  which  will  ultimately  be 
able  to  offer  predictions  about  the  relationships  between  task  demands,  task  performance, 
affect  and  their  impact  on  the  achievement  of  specific  goals.  This  is  an  innovative  approach 
involving  the  development  of  a  quantitative  model  to  translate  the  qualitative  compensatory 
control  model  of  human  performance  under  stress  and  high  workload  (Hockey,  1997)  into  a 
computational  form.  The  compensatory  control  model  proposes  that  physiological  and  affective 
responses  are  inherent  in  the  regulatory  process. 

Affective  state  is  considered  to  be  an  integral  aspect  of  a  person's  response  to  increased  task 
demands.  Traditionally,  psychological  models  have  been  based  on  verbal  descriptions  of  the 
relationships  between  individual  factors,  including  affect,  that  influence  the  human  response  to 
a  given  set  of  task  demands.  These  models  allow  qualitative  descriptions  of  human 
performance,  but  do  not  allow  quantitative  assessment  to  be  made.  The  central  innovation  of 
the  CAWM  approach  is  in  the  explicit  representation  of  the  variables  within  the  compensatory 
control  model  as  mathematical  functions  which  will  enable  testing  to  determine  their  precise 
relationships.  This  application  of  AST  is  discussed  further  in  Section  7.3  of  the  following 
discussion  paper. 


Affective  Skill  Theory  (AST):  A  new  explanation  for  the  role  of  affect  in 
evaluating  performance 


Abstract 


This  paper  proposes  a  new  theory  to  account  for  the  role  of  affect  in  understanding  and 
evaluating  performance  under  stress.  Although  affect  is  recognized  as  a  determinant  of 
performance  there  is  no  universal  agreement  on  its  conceptualization  or  role  in  perfonnance.  In 
Affective  Skill  Theory  (AST),  affect  is  conceptualized  as  a  skill  that  can  be  trained  prior  to  an 
operation.  As  a  skill  affect  can  also  be  used  as  a  key  variable  in  evaluating  operation  success 
based  on  the  ability  to  assess  the  match  between  the  affective  skills  of  personnel  with  operational 
affective  requirements.  The  theory  has  the  potential  to  be  highly  valuable  in  supporting 
operational  planning  through  identifying  situations  in  the  field  which  will  impose  high  affective 
task  demands  on  personnel  and  support  assessment  of  the  optimal  affective  level  of  expertise 
required  of  personnel  to  successfully  complete  the  operation.  This  model  of  human  performance 
conceptualizes  affect  as  an  integral  component  of  an  individual’s  professional  skills  base  and  in 
the  same  way  as  a  cognitive  or  physical  skill  it  contributes  to  whether  a  task  can  be  successfully 
completed  under  high  stress  conditions.  The  proposal  that  affect  can  be  evaluated  as  a  skill  was 
explored  during  a  preliminary  pilot  study  during  which  participants  affective  states  where 
evaluated  undertaking  a  driving  simulation.  During  the  study  preliminary  technical  work  was 
undertaken  which  forms  the  basis  of  future  work  to  develop  an  objective  tool  for  measuring 
individual  affective  skill  levels  during  simulation  training.  The  pilot  and  associated  theoretical 
work  contribute  to  the  literature  on  performance  and  emotion  by  conceptualization  and 
subsequent  measurement  of  affect  as  a  skill  which  enables  successful  task  completion. 


1.  Introduction 


Active  cognition  during  complex  operations  is  critical  component  for  success  in  military 
operations.  Objective  judgment  and  decision-making  are  crucial  operational  functions  which  rely 
on  higher-order  cognitive  pathways  to  process.  Emotions  associated  with  stress  such  as  fear  and 
anger  unless  they  are  controlled  can  overwhelm  prefrontal  cognitive  processes  (Russo  et  ah, 
2005).  The  ability  to  control  emotion  varies  across  personnel  and  often  relates  to  experience.  An 
expert  can  be  viewed  as  having  developed  over  time  and  through  experience  the  level  of  skill 
required  to  control  his/her  emotions  during  high  stress  operations. 


A  significant  amount  of  research  has  been  published  over  the  past  4  decades  concerning  the 
mechanisms  that  underlie  expert  performance  (eg  Chi,  Glaser  &  Farr,  1988,  Ericsson  et  al,  2006, 
Lfot  et  al,  2009).  From  a  practical  perspective,  understanding  expertise  is  important  for  the 
design  of  training  programs  and  from  a  theoretical  perspective,  the  principles  and  mechanisms 
proposed  to  underlie  expertise  can  be  used  to  evaluate  the  generalizability  of  broader  theories 
about  basic  cognitive  processes  and  capacities  and  thus  explain  human  perfonnance  more 
generally”  (Loft  et  al,  2009). 


Affect  is  recognized  as  a  determinant  of  performance  (Gee,  2010).  However  generally  studies 
have  examined  the  influence  of  goal  striving  on  affective  states  (Foo  et  al,  2009,  Koestner  et  al, 
2002;  Louro  et  al.,  2007),  rather  than  the  ability  of  affect  to  influence  goal  striving.  Ballard 
(2009)  found  that  perfonnance  during  a  two-minute  time  period  was  positively  related  to 
affective  state  at  the  end  of  that  time  period.  Reinforcement  sensitivity  theory  (Gray  1990) 
predicts  perfonnance  will  effect  tense  and  energetic  emotional  arousal  most  strongly.  Control 
theory  (Seo  etal,  2004)  makes  predictions  on  what  effect  performance  progress  will  make  on 
affect.  For  example,  when  individuals  experience  activated  states,  such  as  stress  or  alertness, 
they  subsequently  try  harder.  In  summary  these  theories  suggest  when  making  good  progress 
toward  a  goal  an  individual  experiences  positive  affect  such  as  happiness  and  alertness;  while 
making  poor  progress  results  in  negative  affect  such  as  stress  (Ballard  p40,  2009).  Each  of  these 
theories  looks  at  the  effect  of  perfonnance  on  emotion. 


Ballard  (2009)  however  in  addition  to  investigating  the  effect  of  performance  on  affect,  also 
examined  the  effect  of  affect  on  effort  during  goal  striving.  Ballard  found,  “when  an  individual  is 
setting  a  goal,  feelings  such  as  alertness  or  brightness  can  lead  them  to  engage  further  and  set  a 
more  difficult  goal.  On  the  other  hand,  feelings  of  anxiety  or  stress  can  lead  them  to  avoid 
difficulty  by  setting  a  goal  that  is  easier  to  attain.  However,  this  process  changes  during  goal 
striving,  because  once  the  individual  has  committed  to  a  goal,  the  goal  becomes  more  difficult  to 
change.  As  such  when  an  individual  is  striving  toward  their  goal,  they  may  respond  to  feelings  of 
anxiety  or  stress  by  increasing  their  effort  in  order  to  regain  their  progress  toward  the  goal  (2009; 
p47)  ”  SUggCSLS  tq,aL  m  fact  affect  can  be  an  influential  force  within  effective  performance 
rather  than  merely  being  a  viewed  as  an  output  that  is  influenced  by  how  well  or  not  an 
individual  is  performing  in  tenns  of  progress  toward  their  goal. 


Considerable  evidence  shows  that  achievement  goals  and  affect  are  intricately  related.  However, 
within  the  goals  literature  researchers  have  conceptualized  affect  as  both  an  outcome  of  goal 


pursuit  (eg.  Pekrun,  Elliot  &  Maier,  2006)  and  as  an  antecedent  of  goal  adoption  (eg  Elliot  & 
Thrash,  2002).  Thus  again  affect  is  viewed  sometimes  as  an  input,  it  influences  goal  adoption, 
but  also  as  an  output,  it  is  an  outcome  of  goal  pursuit.  Several  models  detailing  possible 
relationships  between  goals,  affect  and  performance  have  been  proposed  (eg  Linnenbrink  & 
Pintrich,  2002;  Pekrun  et  al.,  2006). 


Daniels,  Stupnisky,  Pekrun  et  al,  (2009,  p954),  proposed  a  sequential  model  in  which  initial 
affective  experiences  predict  goals,  goals  predict  discrete  emotions,  and  discrete  emotions 
predict  achievement.  By  design,  the  model  suggests  that  the  relationships  between  constructs  that 
occur  early  in  the  model  (ie,  hopefulness  and  helplessness)  with  those  that  come  later  in  the 
sequence  (ie.  Enjoyment,  boredom,  anxiety)  are  at  least  partially  mediated  by  the  interceding 
constructs  (ie  mastery,  performance  goals).  So  again,  a  mix  of  conceptualizing  affect  as  an  input 
and  an  output  of  performance. 


A  model  which  locates  affect  in  the  context  of  mental  workload  is  the  Compensatory  Control 
Model  (Hockey,  2005).  This  model  recognizes  that  physiological  and  affective  responses  are 
inherent  in  the  regulatory  process.  For  example  the  model  explains  anxiety  as  an  output  of  an 
individual’s  action  monitor.  Affect,  in  this  example  is  a  result  of  the  appraisal  that  current  goals 
are  not  being  met.  When  making  predictions  about  performance  under  stress  and  high  workload 
the  model  describes  positive  affect  as  decreasing  when  task  demands  increase.  There  is  no  firm 
prediction  about  negative  affect  under  high  task  demands,  but  it  suggests  negative  affect  is  most 
likely  to  increase.  Thus  an  individual’s  adoption  of  an  approach  or  avoidance  orientation  may 
influence  the  expected  affective  output  of  performance  goal  striving  (Fletcher,  2010). 


The  limitation  of  this  model  and  each  of  those  outlined  earlier  is  the  limitations  they  impose  on 
themselves  through  restricting  predictions  of  how  higher  task  demands  will  effect  affect  rather 
than  how  individual  affective  skills  can  or  will  effect  goal  commitment  and  achievement.  The 
Affective  Skill  Theory  (AST)  provides  an  important  new  direction  in  theory  through  changing 
the  conceptualization  of  affect  in  mental  workload  models,  models  of  goal  achievement  and  goal 
striving  to  examining  affect  in  terms  of  what  this  variable  brings  to  the  relationship  between 
individual  appraisal  of  workload  and  ultimate  achievement  of  goals.  Conceptualizing  affect  as  a 
skill  that  is  brought  by  an  individual  to  their  work  environment  and  used  to  achieve  high 
performance  outcomes,  as  opposed  to  conceptualizing  affect  as  an  output  influenced  by 
performance,  provides  a  theoretical  model  from  which  prediction  of  a  professional’s  likelihood 
of  goal  achievement  via  measureable  affective  skills  can  be  made. 


2.  Affective  Skills  Theory 


The  work  of  Damasio  (1995)  in  neuroscience  has  convincingly  demonstrated  the 
interdependence  between  emotions  and  activities  previously  considered  to  require  only  rational 
thought,  such  as  problem-solving  and  decision-making.  In  Lazarus’s  (1966)  transactional  model 
of  stress,  people  appraise  task  demands  in  terms  of  their  own  resources  to  cope.  A  professional 
needs  to  perceive  they  have  the  individual  affective  resources  to  cope  with  their  workload.  This 
perspective  support  the  conceptualized  of  affect  as  a  further  resource  that  can  act  as  an  input  to 
support  cognitive  processing.  While  the  role  of  affect  has  been  increasingly  recognized  in 
psychology,  few  theoretical  models  address  the  concept  of  affect  that  contributes  to  whether  a 
task  can  be  successfully  completed  under  high  stress  conditions.  Primarily  affect  has  been 
conceptualized  as  an  output  of  too  many  task  demands  such  as  feelings  of  anxiety  being  the 
result  of  work  overload. 


At  the  same  time  as  models  of  affect  conceptualize  the  variable  as  an  output,  simulation  training 
for  professionals  working  in  high  stress  environments  have  been  training  individuals  to  learn  to 
control  and  use  their  emotions  to  facilitate  optimal  performance  for  many  years.  If  one  is  going 
to  harness  the  strengths  of  affect  to  provide  more  effective  training  and  increase  the  likelihood  of 
success  in  operational  settings  characterized  by  high  task  demands,  then  theoretical  models  must 
go  beyond  conceptualizing  affect  as  merely  a  result  or  output  of  high  task  demands.  Emotion 
prepares  individuals  to  respond  to  eliciting  stimuli  by  coordinating  a  system  for  responses:  anger 
prepares  body  to  fight,  and  fear  prepares  it  for  flight  (Matsumoto  &  Wilson,  2008).  Simulation 
training  works  on  the  premise  that  a  professional’s  confidence  in  his/her  affective  skills  can  be 
strengthened  through  training  whereby  they  have  been  given  the  opportunity  to  experience 
mastery  while  in  a  highly  aroused  state,  for  example  they  have  been  able  to  work  through  fear 
and  successfully  translate  it  into  high  goal  performance.  Additionally,  emotions  help  individuals 
identify  associations  between  event-based  triggers  and  behavioral  consequences  so  as  to  be  better 
equipped  in  similar  situations  in  the  future  (Matsumoto  &  Wilson,  2008).  As  a  resource  that 
supports  identification  of  prior  experiences,  emotions  are  clearly  a  resource  that  can  be  brought 
to  bear  by  individuals  either  in  the  training  or  operational  context. 


The  AST  approach  proposes  that  emotions  that  typically  strengthen  and  support  performance  are 
‘affective  skills’  and  that  these  skills  are  as  integral  to  performance  as  technical  and  cognitive 
skills.  Affective  skills  can  be  understood  as  new  addition  o  the  non-technical  skills  set.  The 
current  project  commences  the  work  to  develop  a  construct  definition  for  ‘affective  skills’  and 
build  a  conceptual  framework  which  describes  how  it  is  used  to  successfully  attain  professional 


goals.  This  construct  will  cover  awareness  and  control  of  emotion  across  both  arousal  and 
valence. 


Affective  skills  are  a  concept  that  has  been  recognized  in  the  field  of  professional  sport  for  some 
time.  Affective  Intelligence  is  a  subscale  of  a  tool  developed  to  measure  ‘mental  toughness’  in 
athletes.  Affective  intelligence  recognizes  that  optimal  perfonnance  on  the  sports  is  dependent 
on  an  athlete  being  able  to  remain  in  control  of  their  emotions,  no  matter  what  obstacles  they 
encounter  and  be  able  to  be  actively  able  to  bring  their  emotions  into  play  to  facilitate  optimal 
performance  (Gordon  &  Gucciardi,  2010;  Gucciardi  &  Gordon,  2009;  Connaughton  et  ah,  2008). 


Some  research  has  reported  that  motivation  to  achieve  can  only  be  positively  influenced  by 
emotions  that  are  perceived  to  be  ‘positive’.  Positive  affective  experiences  facilitate  the  retrieval 
of  positive  self-  and  task-related  information,  whereas  negative  affective  experiences  facilitate 
the  retrieval  of  negative  self-  and  task-related  infonnation  (Linnenbrink  &  Pintrich,  2002; 

Daniels  et  al,  2009).  However,  unpleasant  activated  states  have  also  been  shown  to  be  motivating 
during  goal  striving  because  they  lead  individuals  to  mobilize  effort  (Ballard,  2009).  There  are 
likely  to  be  situations  in  which  too  much  unpleasant  activation  is  detrimental  to  motivation. 

There  will  be  individual  differences  in  how  much  unpleasant  activation  is  too  much  before 
individuals  start  to  lose  motivation.  It  has  always  been  a  goal  of  simulation  training  to  gradually 
expose  trainees  to  increasing  levels  of  stress  until  they  are  able  to  tolerate  a  significantly  higher 
stressful  state  before  they  start  to  withdraw  effort.  The  goal  of  training  is  to  provide  the 
opportunity  for  trainees,  through  repetition,  to  prove  to  themselves  that  they  can  maintain 
optimum  performance  under  highly  stressful  conditions  and  that  they  do  not  need  to  withdraw 
but  they  can  use  their  emotions  to  support  their  own  perfonnance. 


Conceptualizing  affect  as  a  skill  that  plays  an  important  role  in  perfonnance  also  recognizes  its 
role  in  supporting  and  strengthening  cognitive  and  technical  skills.  If  anxiety  and  fear  are  not 
controlled  and  used  to  support  performance  they  inevitably  degrade  and  undermine  performance 
both  at  a  technical  and  cognitive  skills  level.  The  AST  approach  could  provide  the  basis  for  the 
development  of  a  model  that  can  predict  the  effect  of  different  affective  states  on  cognitive 
processing  during  high  task  demands.  For  example,  when  conceptualized  as  an  input,  affect  can 
be  investigated  to  determine  the  optimal  degree  of  affective  intensity  to  support  decision  making 
under  stress. 


In  military  operations  the  potential  usefulness  of  the  AST  approach  would  be  in  the  assessment 
of  combat  affective  readiness  and  in  the  prediction  of  affective  performance.  The  relationships 
between  affect,  cognition  and  performance  support  further  predictions  of  overall  performance. 
The  ability  of  monitoring  techniques  to  predict  overall  cognitive/affective  readiness  before  an 
actual  operation  can  provide  useful  logistical  information  regarding  fitness  for  duty  (Russo  et  al, 
2005).  If  decrements  are  identified  planning  can  be  undertaken  to  more  closely  align 
cognitive/affective  reserves  with  essential  tasks  or  training  targeting  shortfalls  can  be  undertaken 
if  prior  to  operational  undertaking. 


3.  Training  Affective  Skills 


There  are  many  advantages  to  being  able  to  train  a  professional  to  improve  their  affective  skills. 
Military  reports  indicate  that  the  overwhelming  majority  of  combat  personnel  experience  fear 
before  and/or  during  battle.  Often  physical  symptoms  associated  with  fear  are  reported  such  as 
nausea  and  heart  palpitations.  Although  fear  is  natural  and  often  important  to  survival  it  can’t  be 
allowed  to  manifest  to  the  extent  that  it  interferes  with  and  reduces  combat  effectiveness  (Redden 
et  al,  p565). 


Keeping  a  clear  head  in  time  of  crisis  is  critical.  Many  professionals  across  a  multitude  of  high- 
stress  domains  including  the  military,  police  and  accident  response  teams  face  such  situations. 
Training  higher-order  mental  abilities  now  most  often  involves  computer  driven  simulators.  The 
advantage  of  simulators,  which  have  been  in  widespread  use  since  the  early  1980s,  is  that  they 
allow  trainees  to  practice  extreme  worst  case  scenarios.  While  reinforcing  important  technical 
skills  simulators  are  also  teaching  trainees  something  more  important:  how  to  draw  on  an  optimal 
blend  of  reason  and  emotion.  They  leam  how  to  ignore  their  fear  when  fear  isn't  useful  and  how 
to  make  quick,  complicated  decisions  in  the  most  fraught  situations  (Lehrer  J,  2009). 
Psychological  skills  training  programs  teach  techniques  and  strategies  used  to  assess,  monitor 
and  adjust  thoughts  and  feelings  to  produce  mental  states  that  enhance  performance  (Vealey, 
2007;  Gordon  &  Gucciardi,  2010).  Simulators  provide  the  opportunity  to  put  these  skills  to  the 
test  through  the  opportunity  to  practice  the  achievement  of  goals  while  under  stress. 


Soldiers  must  make  complicated  battlefield  decisions;  however,  no  matter  how  difficult  or 
unprecedented  the  problem,  a  highly  trained  professional  has  the  ability  to  look  past  primal 
emotions  and  carefully  think  about  how  to  proceed  (Lehrer,  2009).  All  indications  are  that  there 
does  in  fact  exist  a  skill  set,  best  described  as  affective  skills,  that  professionals  are  aware  of 


within  themselves  and  able  to  bring  to  bear  to  achieve  high  demand  goals.  In  aviation,  pilots 
have  long  been  taught  to  practice  staying  calm  (Lehrer,  2009).  They  are  able  to  do  this  through 
the  opportunity  simulators  provide  to  practice  building  confidence. 


However,  despite  the  aviation  industry’s  success  in  achieving  continual  improvements  in  safety, 
the  identification  of  Non-Technical  skills  (NTS)  as  a  root  cause  in  many  accidents  has  remained 
a  problem.  It  is  clear  that  the  industry  is  making  attempts  to  address  this  problematic  situation, 
as  seen  by  the  push  to  implement  NTS  training  and  assessment  to  counter  the  unfortunate  effects 
of  NTS  deficits.  However,  the  assessment  of  NTS  is  still  seen  to  be  challenging.  No  model  or 
framework  has  yet  been  devised  that  is  able  to  be  effectively  used  in  practice. 


Mavin  (2010)  through  an  investigation  of  the  criteria  used  to  assess  pilots’  performance 
examined  individual  check  captain’s  personal  evaluation  criteria.  Significantly,  check  captains 
identified  that  the  first  element  of  the  decision  making  process  was  having  the  confidence  to  be 
able  to  make  decisions.  A  candidate  could  fail  if  they  were  viewed  as  being  unable  to  take  the 
final  step  in  taking  responsibility  or  demonstrated  uncertainty  in  decision-making.  The  ability  to 
take  charge  of  the  situation  was  the  first  important  step  for  command.  The  pivotal  and  essential 
role  affective  skills  play  in  this  ability  is  demonstrated  in  the  following  quote:  I’ve  always 
believed  that  flying  aero  planes  is  nothing  more  than  an  exercise  in  self-confidence  (Mavin,  2010 
plOO). 


4.  Measuring  Affect 


In  research  which  evaluates  simulator  training  of  cognitive  skills  such  as  situation  assessment 
and  decision-making,  affective  state  has  therefore  become  a  key  variable  of  interest  (Tichon, 
2010).  However,  as  has  been  the  experience  in  aviation,  across  all  professional  domains  the 
assessment  of  NTS  is  challenging. 


The  current  AOARD  project  (104011)  was  devised  to  identify  a  reliable  set  of  objectively 
measurable  features  and  responses  that  highly  correlate  to  target  emotions  via  eye-tracking, 
facial  feature-extraction  and  electromyography  (EMG)  technologies.  In  trying  to  achieve  this  it 
has  become  apparent  that  existing  theories  have  limits  in  terms  of  explaining  the  link  between 
affect  and  performance.  The  relationship  between  these  two  variables  does  not  seem  to  have 


been  previously  investigated  at  all  within  the  context  of  affectively  intense  simulator-based 
training  settings. 


All  studies  on  affect  and  perfonnance  discussed  earlier  in  this  paper  are  characterized  by  one 
major  limitation.  That  is,  that  all  research  is  currently  constrained  by  the  necessity  of  relying  on 
subjective  self-report  measures  to  measure  affect.  While  technical  performance  can  be  measured 
objectively,  to  date  affect  cannot. 


Subjective  measures  of  stress  include  interviews,  open-ended  and  scaled  response  questionnaires. 
“Subjective  measures  are  particularly  useful  as  a  means  to  assess  person-environment  ‘fit’  based 
on  direct  subjective  report  of  stressful  experience.  However,  great  caution  should  be  exercised  in 
using  subjective  self-report  measures  to  assess  stress  in  other  domains  and  purposes.  For 
example,  if  a  researcher  is  interested  in  evaluating  stress  produced  by  a  particular  situation, 
subjective  measures  may  be  misleading.  An  environment  perceived  as  stressful  by  one  individual 
may  not  be  perceived  as  stressful  to  another,”  (Redden  et  al,  2004;  p554). 


When  monitoring  the  effects  of  stress  the  military  prefer  objective  measures.  Currently 
physiological  measures  are  preferred  because  most  are  fairly  simple  to  obtain,  easily  quantified, 
and  generally  more  objective  than  either  self-report  or  behavioral  ratings.  For  example  salivary 
cortisol  and  salivary  amylase  provide  a  quick,  non-intrusive  and  inexpensive  means  to  predict 
plasma  catecholamine  levels  (e.g.  norepinephrine)  under  stressful  conditions  (Chatterton  et  al., 
1996).  The  Army  Research  Laboratory’s  salivary  amylase  collection  protocol  is  simple  and  easy 
enough  to  be  managed  in  a  field  setting  (Blewett  et  al.,  1994).  Heart  rate  and  blood  pressure  are 
also  simple,  quick  and  useful  metrics  to  study  or  monitor  the  effects  of  stress  (Redden  et  al, 
p553).  The  Army  when  assessing  measures  for  use  in  the  field  obviously  look  for  those  that 
offers  a  combination  of  being  lightweight,  easily  transportable  and  quickly  and  easily  obtained 
from  participants. 


Simulators  provide  the  advantage  of  being  able  to  replicate  high  stress  operational  environments 
safely  and  realistically.  Within  these  replicated  environments  the  opportunity  arises  to  test  and 
develop  more  advanced  measures.  Contemporary  research  into  the  effects  of  mental  and  physical 
workload  refer  to  psycho  physiological  measures  such  as  electroencephalography  (EEG),  cardiac 
changes  (heart  rate,  blood  pressure),  ocular  events  (number  and  duration  of  eye  blinks),  changes 
in  skin  response,  muscle  activity  (electromyography:  EMG)  and  respiration  (Redden,  2004). 
Increasingly  it  is  being  recognized  that  capturing  behavioral  data  from  participants  such  as  facial 


expressions  or  head  movements  may  be  a  more  accurate  representation  of  how  and  what  they 
feel  and  a  better  alternative  to  self-report  questionnaires  that  are  not  only  subject  to  bias  but  also 
interrupt  participant’s  affective-cognitive  processes  (Reynolds  &  Picard,  2004;  Ahn  et  al  2010). 


Feature  extraction  as  an  objective  measurement  tool  for  affect  recognition  has  been  garnering  the 
most  significant  increase  in  interest  (Liao  etal,  2006;  Bailenson  et  ah,  2008).  Stress  has  been  a 
key  variable  of  interest  with  investigations  revealing  features  are  potentially  sensitive  and  robust 
to  stress  (Liao  et  al.,2006),  however  a  model  using  physiological  measures  in  tandem  with  facial 
tracking  was  found  to  be  more  reliable  than  one  relying  on  either  cue  alone  (Bailenson,  2008). 

Bailenson’s  (2008)  model  was  developed  using  videotapes  of  subjects’  faces,  from  which  trained 
coders  made  second -by-second  assessments  of  sadness  or  amusement,  in  conjunction  with 
physiological  measurements,  including  cardiovascular,  somatic  and  electrodermal  responses,  to 
predict  emotion.  Learning  algorithms  were  then  used  to  link  facial  anchor  points,  as  determined 
by  NEVEN  vision  which  had  53  facial  features,  and  physiological  responses  to  emotional 
responses  rated  by  the  coders.  From  these  data  sources  the  researchers  developed  an  accurate 
model  of  how  emotions  were  expressed  in  response  to  the  stimuli  under  study. 


El  Kaliouby  &  Robinson  (2005)  developed  a  general  computational  model  for  facial  affect 
inference  and  have  implemented  it  as  a  real-time  system.  This  approach  used  dynamic  Bayesian 
networks  for  recognizing  six  classes  of  complex  emotions.  Their  experimental  results 
demonstrated  that  it  was  more  efficient  to  assess  a  human’s  emotion  by  looking  at  the  person’s 
face  historically  over  a  two  second  window  instead  of  just  a  current  frame.  Their  system  was 
designed  to  classify  discrete  emotional  classes  as  opposed  to  the  intensity  of  each  emotion. 


Yoshie  et  al  (2008)  examined  the  relationships  among  psychological  stress,  electromyographic 
(EMG)  activity,  and  perfonnance  in  pianists.  EMG  activity  was  reported  as  a  reliable,  objective 
measurement  of  the  underlying  target  construct,  emotion.  In  response  to  stress,  both  agonist  and 
antagonist  muscle  activity  increase  resulting  in  co-contraction  and  increased  joint  stiffness. 
Elevated  muscle  activity  associated  with  psychological  stress  also  results  in  increased  force 
outputs  and  can  lead  to  deterioration  in  overall  signal-to-noise  ratio  in  the  motor  control  system 
resulting  in  further  observable,  recognizable  patterns  of  motor  performance  (Yoshie  et  al,  2008). 


The  most  recent  work  investigating  applications  of  automatic  facial  detection  indicates  that 
future  behavior  can  be  successfully  predicted  based  solely  on  facial  expressions.  Facial 
expression  recognition  is  a  bottom-up  approach  in  which  the  correlation  between  facial 


movement  and  a  particular  output  (a  behavior  or  an  emotion)  becomes  the  formula  for 
establishing  not  just  a  classification  model  but  a  prediction  model  as  well.  Based  only  on  the  raw 
data  of  facial  feature  movements,  models  have  been  developed  which  successfully  classify 
different  emotions  and  were  subsequently  able  to  predict  whether  a  consumer  would  purchase  a 
product  or  not  and  predict  whether  a  driver  would  be  involved  in  an  automobile  accident  before 
it  happened  (Ahn  et  ah,  2009). 


In  terms  of  published  studies  on  evoked  emotions,  research  in  this  area  has  been  limited  by  its 
reliance  on  individual  subjective  coders  and  the  use  of  passive  non-interactive  movies  to  evoke 
emotion  in  participants  (Bailenson,  2008).  Immersion  in  replicated  high  stress  operational 
environments  via  simulation  provides  a  far  superior  research  opportunity  for  progressing 
investigation  into  feature  extraction  and  its  correlation  to  affective  states.  The  current  research 
aims  to  systematically  explore  a  combination  of  feature  extraction,  EMG  and  physiological 
measures  as  the  first  stage  in  the  development  of  an  objective  measurement  approach  to  evaluate 
learner  affective  states  during  complex  cognitive  training  via  simulation. 


5.  METHODS  -  PILOT  TEST  ONE 


A  pilot  study  was  conducted  in  May  2010.  This  pilot  was  undertaken  in  a  university  laboratory 
setting  to  achieve  proof  of  concepts  and  physical  equipment  trialing.  The  objectives  were  two¬ 
fold:  (i)  test  the  hardware  and  software  setup  with  the  aim  of  ironing  out  any  problems,  and  (ii) 
to  obtain  simultaneous  physiological  and  subjective  affect  measures.  Although  the  data  was  used 
to  probe  for  correlations  between  the  physiological  state  and  reports  of  emotional  state,  the 
primary  aim  of  the  first  pilot  was  to  check  all  programming  requirements  had  been  undertaken  to 
ensure  all  monitoring  equipment  was  functioning  as  required  by  the  study  and  were  appropriately 
synchronized  with  each  other.  Additionally  it  provided  a  trial  set  of  data  for  filtering  and  making 
initial  analytical  sweeps  to  assist  determine  final  analysis  requirements.  A  second  Pilot  Test  was 
then  to  be  undertaken  with  trainee  commercial  pilots  in  a  full  flight  simulator  with  this  second 
round  of  data  being  primarily  used  to  identify  initial  physiological  indicators  of  emotional  state. 
Unfortunately,  having  submitted  Human  Use  Documentation  seeking  approval  to  use  the  flight 
simulator  on  27th  May  2010  the  approvals  had  not  been  received  by  the  end  of  the  12  month 
project  time.  Therefore  Pilot  Test  One  only  will  be  reported  here. 


In  Pilot  Test  One  muscle  activity  is  recorded  by  electromyography  (EMG)  and  feature  extraction 
via  eye  responses  using  an  eye-tracker.  The  study  used  a  driving  simulation  in  which  subjects 


had  to  steer  through  a  sequence  of  obstacles.  The  test  consisted  of  four  road  segments.  After 
completing  each  segment  the  simulation  was  halted  and  the  user  was  required  to  fill  in  a 
questionnaire  assessing  their  affective  state. 


5.1  Participants 


Volunteers  were  recruited  from  the  post-graduate  student  and  staff  population  at  the  University 
of  Queensland’s  School  of  Human  Movement  Studies.  Prerequisites  for  participation  were 
nonnal  or  corrected-to-normal  vision  and  being  licensed  to  drive  on  Australian  roads.  Five 
subjects  took  part  in  the  study  ranging  in  age  from  26  to  5 1  years. 


5.2  Apparatus 


5.2.1  Simulator 

The  experiments  were  conducted  in  a  fixed-base  driving  simulator,  which  consists  of  a  Silicon 
Graphics  (SGI)  Onyx  3200  graphics  system,  Barco  808S  CRT  projector  and  Logitech  MOMO 
force-feedback  steering  wheel.  The  steering  wheel  is  controlled  by  a  Dell  Pentium  4  PC,  which 
shares  a  high-speed  ethernet  connection  with  the  graphics  computer.  Scenes  are  rendered  by 
custom  OpenGL  Performer  software  and  front-projected  onto  a  flat,  matte  white  surface  3.4m 
wide  by  2.7m  high,  which  subtends  78  by  68  degrees  of  arc  at  a  viewing  distance  of  2.54m. 
Vertical  update  and  software  update  rates  are  fixed  at  72Hz,  and  pixel  resolution  is  1280x1024 
with  32-bit  color  depth.  The  steering  wheel  turns  through  240  degrees  (-120deg  to  +120deg)  and 
produces  a  return  force  proportionate  to  angular  rotation.  Steering-wheel  angle  is  sampled  at 
each  vertical  update. 


5.2.2  Virtual  Scene 

A  virtual  environment  consisting  of  a  winding  road  was  used  for  the  pilot  test  simulation.  The 
roadway  generated  in  each  of  the  four  segments  of  a  single  test  consisted  of  7  continuously 
linked  sections.  The  order  (and  direction)  in  which  each  curve  was  encountered  was  randomized 
on  a  trial-by-trial  basis.  Randomization  of  curve  magnitude  and  direction  were  intended  to 
eliminate  systematic  sources  of  variability  in  performance.  Figure  1  shows  views  of  the  roadway 
from  the  participants’  point-of-view  encountering  four  different  levels  of  obstacle  obstruction. 


5.2.3  Electromyography  (EMG) 


EMG  data  were  recorded  at  1000Hz  using  surface  electrodes  feeding  into  a  National  Instruments 
analogue  recorder  that  is  controlled  by  a  LabView  program  running  on  a  central  PC  (Control 
PC).  Eight  synchronous  analogue  recording  channels  were  available. 

The  activity  of  the  following  muscles  were  of  interest  and  were  recorded: 

1.  Jaw  clenching  ( masseter ). 

2.  Shoulder  tension  muscles  ( upper  Trapezius). 

3.  Wrist  flexing. {Flexor  carpi  radialis). 

4.  Calf  tension  ( Lateral  gastocnemius). 

5 .  Thigh  tension  ( Vastus  lateralis). 

It  was  our  intention  to  also  record  eyebrow  movement  ( Corrugator  supercilii )  but  it  proved  too 
difficult  to  attach  electrodes  effectively  without  interfering  with  the  eye-tracker  head  harness. 


5.2.4  Feature  Extraction  via  Eye  tracking  and  Pupilometry 

Eye  movement  and  pupil  size  data  were  be  recorded  using  an  SR  Research  EyeLink  II  head- 
mounted  tracker,  perfonning  binocular  500Hz  sampling.  The  eye  tracker  is  connected  to  a 
dedicated  PC  (the  EyeLink  ‘host  PC’)  that  has  custom  hardware,  on  a  PCI  card,  to  communicate 
with  the  eye  tracker  and  collect  its  data.  The  EyeLink  II  system  also  requires  a  second  “display 
PC”  which  manages  interaction  with  the  subject  using  the  eye  tracker.  The  host  PC  effectively 
acts  as  a  peripheral  to  the  display  PC. 


Data  collected  by  the  eye-tracker  includes  relative  pupil  size,  eye  position  and  movement 
velocity.  The  final  data  includes  some  derived  “events”:  blinks,  fixations  and  saccadic 
movements  that  are  calculated  by  the  eye-tracker  host  before  it  sends  the  data  file  to  the  display 
PC.  This  data  file  is  accumulated  on  the  host  PC  and  transmitted  at  the  end  of  a  session. 


The  eye  tracker  requires  an  initial  calibration  procedure  for  each  participant.  During  this  the 
subject  is  fitted  with  the  head  harness,  and  then  responds  to  fixed  stimuli,  enabling  the  eye 
tracker  software  to  calculate  how  to  extrapolate  from  eye  movement  and  position  to  gaze 
direction  for  the  individual  subject. 


Feature  measures  which  were  derived  from  the  eye-tracking  data  were:  Blinking  Frequency 
(BF),  Average  Eye  Closure  Speed  (AECS),  Percentage  of  Saccadic  Eye  Movement  (PerSac), 
Gaze  Spatial  Distribution  (GazeDis),  Percentage  of  Large  Pupil  Dilation  (PerLPD), 


5.2.5  Survey 

A  self-report  survey  was  used  in  this  initial  stage  to  gain  an  indication  from  participants  of  those 
emotions  that  were  engendered  by  the  test  simulation.  A  suitable  general  measure  of  emotional 
states  was  ascertained  to  be  the  Multiple  Affect  Adjective  Checklist  (MAACL-R)  which  has 
been  extensively  used  in  the  investigation  of  the  impact  of  stress  on  psychological  functioning 
(Hunsley,  1990)  and  is  currently  in  use  in  simulator  training  evaluations.  In  studies  of  acute 
stress,  the  Army  Research  Laboratory  (ARL)  has  found  that  temporary  stress  effects  such  as 
anxiety,  depression  and  hostility  are  revealed  by  the  Multiple  Affect  Adjective  Checklist-Revised 
(MACCL-R).  The  checklist  consists  of  132  adjectives  that  comprise  five  primary  subscales 
(anxiety,  depression,  hostility,  positive  affect,  and  sensation  seeking).  The  checklist  can  be 
completed  in  approximately  five  minutes  (Redden  et  al,  2004).  The  drawback  of  the  scale  is  that 
it  more  easily  achieves  independence  between  positive  and  negative  affect  rather  than  between 
two  dimensions  of  negative  affect,  however  the  goal  is  to  use  the  items  as  a  starting  point  from 
which  to  commence  the  research  enabling  later  theoretically  derived  dimensions  of  affect  to  be 
identified  enabling  reliable  differentiation. 


5.3.  Data  Synchronization 

In  order  to  investigate  correlation  between  affect  state  and  all  the  physiological  data  the  various 
data  streams  had  to  be  synchronized.  In  this  experiment  the  eye  tracker  data,  recorded  as  a 
sequence  of  data  records  time-stamped  to  an  accuracy  of  2ms,  and  the  EMG  data,  recorded  at  a 
sample  rate  of  1000Hz  per  second,  needed  to  be  synchronized  prior  to  analysis. 


The  eye  tracker  library  has  a  routine  to  set/unset  its  host’s  parallel  port  data  lines,  and  when  it 
does  so  it  logs  the  action  as  a  time-stamped  message  in  its  data  file.  This  mechanism  was  used  to 
achieve  timing  synchronization  between  the  eye  tracker  and  EMG  data  by  routing  one  of  the 
parallel  port  lines  of  the  eye  tracker  host  PC  to  one  of  the  analogue  data  channels  on  the  EMG 
recorded.  Setting  or  dropping  the  TTL  voltage  on  this  line  via  the  eye  tracker  control  program, 

•  sets  a  time  point  in  the  EyeLink  data  via  the  timestamp  of  the  action  message,  and 

•  Records  a  step-change  in  the  value  recorded  by  the  monitoring  channel,  which  is 
aligned  with  the  EMG  data  recorded  on  the  other  channels. 


The  margin  of  error  in  this  mechanism  was  the  delay  between  the  EyeLink  II  host  recording  the 
output  action  message  and  the  change  in  voltage  initiated  by  that  action  reaching  the  analogue 
recorder  and  affecting  its  output.  It  was  judged  that  this  was  not  significant  in  terms  of  the  2ms 
accuracy  of  the  EyeLink  data  itself. 


Figure  1:  Example  scenes  from  the  roadway  a)  Open  road,  b)  Truck 
blocking  lane,  c)  Nighttime/restricted  viewing,  d)  Unexpected  oncoming 
vehicle  while  passing  truck. 
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Figure  2.  Experimental  Set-up 


5.4  Experimental  Setup 


The  experimental  setup  is  shown  in  Figure  2.  It  has  three  separate  components:  scene  generation 
and  display,  EMG  recording,  and  eye  tracker  control  and  recording.  Each  of  these  requires 
coordination  and  synchronization  between  them.  The  initial  decision  to  be  made  when  designing 
the  system  was  where  to  locate  the  central  controller  that  coordinates  these  components.  The  in- 
house  SGI  scene  generation  software  that  was  adapted  for  this  project  was  originally  designed  as 
a  self-contained  experiment  control  program  that  ran  just  on  the  SGI.  Similarly  the  EyeLink  II 
system  came  with  programs  (with  source)  for  running  experiments  that  generated  scenarios  on 
the  PC  monitor  and  captured  only  eye  tracker  data.  Either  could  be  used  as  a  basis  for 
controlling  our  more  complex  experimental  setup.  We  decided  to  extend  the  eye  tracker  control 
software  on  the  display  PC  to  be  the  overall  controller.  This  was  done  because  we  had  adopted 
the  mechanism  of  using  EyeLink  functions  for  synchronizing  the  data  streams,  and  also  because 
future  work  on  the  project  will  be  off-site  where  the  SGI  will  not  be  available. 

The  architecture  of  the  system  is  illustrated  in  Figure  2.  There  are  four  computers  in  the 
complete  system: 

1.  Stand-alone  steering-wheel  control  PC  (operating  system:  Microsoft  Windows  2000). 
This  ran  proprietary  Logitech  software  that  managed  the  steering  wheel  and  passed 
position  infonnation  to  the  scene  generator. 

2.  SGI  scene  controller  (o/s:  IRIX).  Running  the  in-house  scene  generation  and  control 
program. 

3.  EyeLink  II  host  PC  (o/s:  DOS).  Running  proprietary  SR  Research  software  that 
controlled  the  eye  tracker  hardware. 

4.  Experiment  controller  PC  (o/s:  Microsoft  Windows  XP).  This  ran  a  LabView  data 
collection  program,  for  EMG  data,  and  the  experiment  control  program,  which  also 
managed  the  EyeLink  II  display  functions  (via  the  EyeLink  II  library). 


The  experiment  control  program  communicates  with  the  EyeLink  host  and  the  SGI  Scene 
Generator,  and  collects  all  the  data.  The  controller  also  indirectly  manages  the  EMG  recorder.  It 
starts  the  program  via  system  calls,  and  effects  synchronization  by  requesting  that  the  EyeLink 
host  set/drop  its  parallel  port  lines  as  described  above. 


5.4.1  Control  Program 

The  top-level  control  functionality  was  added  to  the  software  that  controlled  the  eye  tracker  data 
recording.  This  software  was  based  on  an  example  program  that  was  supplied  with  the  EyeLink 
II  by  SR  Research.  SR  provided  this  to  enable  experimenters  to  develop  software  to  suit  their 
particular  deployment  situations.  The  supplied  program  handled  initialization  and  interactive 


calibration  of  the  eye  tracker  at  the  start  of  the  experiment,  and  close  down  of  the  eye  tracker  and 
automatic  downloading  of  the  eye  tracker  data  at  the  close. 


This  original  framework  was  extended  to  coordinate  control  of  generated  scenario  with  the  SGI, 
and  also  to  synchronize  with  the  EMG  recorder  at  the  start  and  end  of  each  generated  segment. 
Coordination  with  the  SGI  is  via  TCP/IP  sockets,  which  are  implemented  on  both  Windows  and 
UNIX  (IRIX)  operating  systems.  A  corresponding  socket  interface  was  added  to  the  existing 
SGI  scene  generator  as  a  separate  asynchronous  process. 


5.4.2  Test  Procedure 

The  subject’s  task  involved  steering  along  the  simulated  highway  maintaining  position  in  the 
left-hand  lane  unless  overtaking  or  avoiding  obstacles.  Forward  velocity  was  kept  constant, 
meaning  that  the  subjects  were  not  required  to  regulate  their  speed,  only  to  steer.  Each  trial  was 
initiated  by  the  subject  pressing  a  button. 


Each  test  consists  of  four  segments  and  each  segment  contains  seven  obstacles.  The  sequence 
and  type  of  obstacles  were  fixed.  Each  obstacle  was  one  of  the  following. 

•  Overtaking  a  single  vehicle  ahead 

•  Overtaking  a  single  vehicle  ahead  and  finding  a  second  vehicle  in  front  of  it  which  was 
not  visible  when  commencing  the  maneuver 

•  Overtaking  a  single  vehicle  ahead  and  finding  an  oncoming  vehicle  in  the  other  lane 

•  Negotiating  blocks  strewn  across  the  road 

•  One  of  the  3  overtaking  tasks  with  the  addition  of  blocks  strewn  across  the  road 

•  Finding  the  road  completely  blocked  by  a  truck  broadside-on  -  this  was  only  used  as  the 
last  obstacle  in  the  test 


In  addition  to  the  sequence  of  obstacles  the  difficulty  of  the  task  was  manipulated  by  varying  the 
speed  at  which  the  subject  was  progressing  and  also  by  reducing  the  forward  visibility.  The 
combination  of  obstacle  difficulty,  increasing  speed,  and  reduction  of  visibility  was  used  to  make 
each  of  the  four  segments  more  difficult  than  the  last.  The  first  segment  served  to  familiarize  the 
subject  with  the  basic  task,  the  second  was  more  challenging,  the  third  introduced  reduced 
visibility,  and  the  fourth  ended  with  a  completely  blocked  road. 


Details  of  all  settings  are  included  in  Table  1,  which  also  shows  the  timing  of  segments  and 
obstacles.  Each  segment  lasted  1  minute  38  seconds  so  the  total  driving  time  was  6  minutes  32 
seconds,  interrupted  by  three  pauses  for  the  administration  of  the  MAACL-R  (Multiple  Affect 
Adjective  Checklist-Revised)  which  was  then  completed  for  a  fourth  and  final  time  on 
completion  of  the  driving  simulation. 


Table  1.  The  sequence  of  obstacles  in  the  scenario 


Sequence 

Speed 

Visibility 
Level  (1-8) 

Obstacle  type 

Time  at  which 
obstacle  is  reached 

Start  Trial 

0 

50 

8 

Overtake  car 

11 

50 

8 

Overtake  car 

22 

50 

8 

Overtake  2  cars 

36 

50 

8 

Overtake  truck 

50 

50 

8 

Overtake  2  cars 

1.04 

50 

8 

Overtake  car 

1.18 

50 

8 

Overtake  truck 

1.32 

End  Section  1 . 

1:38 

Subject  fills  in  questionnaire 

Start  Section  2 

0 

60 

8 

Overtake  truck 

8 

60 

8 

Boxes 

22 

60 

8 

Overtake  car 

36 

60 

8 

Overtake  car  with  boxes 

50 

60 

8 

Overtake  truck  with  boxes 

1.04 

60 

8 

Overtake  car  &  meet  oncoming  car 

1.18 

60 

8 

Boxes 

1.32 

End  Section  2. 

1:38 

Subject  fills  in  questionnaire 

Start  Section  3 

0 

70 

7 

Boxes 

8 

70 

6 

Overtake  truck  and  car 

22 

70 

5 

Overtake  car 

36 

70 

4 

Overtake  car  with  boxes 

50 

70 

4 

Overtake  truck  and  car 

1.04 

70 

5 

Overtake  truck 

1.18 

70 

8 

Overtake  car  &  meet  oncoming  car 

1.32 

End  Section  3. 

1:38 

Subject  fills  in  questionnaire 

Start  Section  4 

0 

70 

7 

Overtake  car  &  meet  oncoming  car 

8 

70 

6 

Overtake  truck  with  boxes 

22 

70 

5 

Overtake  truck  and  car 

36 

70 

4 

Boxes 

50 

70 

4 

Overtake  truck  with  boxes 

1.04 

70 

4 

Overtake  truck 

1.18 

70 

4 

Truck  across  road 

1.38 

End  Trial. 

1:38 

Subject  fills  in  questionnaire 

6.  Results 


Previous  research  has  identified  a  number  of  parameters  relating  to  the  eyes  and  their  movement 
which  are  influenced  by  affective  state  and  specifically  state  anxiety.  Blinks,  saccades  and  pupil 
dilation  have  all  been  reported  as  varying  systematically  with  manipulations  to  stress  or 
measured  anxiety  levels.  Chapman  et  al.  (1999)  conditioned  subjects  to  expect  an  electrical 
shock  to  their  finger-tip,  producing  raised  levels  of  anxiety  and  stress.  During  periods  shortly 
before  a  shock,  the  team  recorded  increases  in  the  cycling  of  pupil  size  (i.e.  variability  in  pupil 
size  over  time)  which  they  attributed  to  a  rise  in  anxiety  levels.  Partla  and  Surakka  (2003) 
exposed  subjects  to  images  designed  to  produce  positive  or  negative  arousal  in  subjects  and 
reported  changes  in  the  maximum,  short-term  pupil  dilatory  response  which  they  tenned  PSV 
(Pupil  size  variation).  Some  tentative  links  between  eye-movement  and  affective  state  have  also 
been  reported  for  subjects  observing  static  faces  portraying  a  range  of  emotional  expressions. 
(Susskind  et  al.  2008).  In  particular,  the  authors  report  increases  in  peak  saccade  velocity  in 
response  to  fearful  expressions.  Perhaps  one  of  the  more  widely  investigated  measures  is  blink 
rate.  The  general  consensus  is  that  blinking  increases  as  anxiety  levels  increase  (e.g.  Harrigan  & 
O’Connell,  1996),  however  the  opposite  result  has  also  been  reported  (Liao  et  al.  2006). 


Subject  A:  Eye  movements 
Movements  when  anxious  as  %  of  movements 
when  not  anxious 


#blinks  %sacc  sacc  amplitude  sacc  velocity  pupil  size 

variation 


Three  of  the  five  subjects  tested  in  the  driving  simulator  reported  increasing  levels  of  anxiety  in 
relation  to  increasing  level  of  task  difficulty.  In  the  above  graph  the  results  are  provided  for  one 
subject  recorded  via  eye  tracking  and  pupilometry  who  reported  a  systematic  increase  in  anxiety 
levels  via  the  MAACL-R  as  the  driving  task  progressed.  The  measures  in  each  category  of  eye 
movement  were  averaged  across  periods  of  low  anxiety  to  form  a  baseline  level.  The  figures 
shown  in  the  graph  are  the  measured  response,  in  that  category  of  eye  movement,  during  periods 
of  high  anxiety  relative  to  baseline  (indicated  at  100  on  the  vertical  axis).  The  categories  listed 
are: 


•  #blinks:  total  number  of  blinks  during  trials  in  which  anxiety  was  reported. 

•  %sacc:  percentage  of  time  spent  making  saccadic  eye  movements  during  trials  in  which 
anxiety  was  reported. 

•  sacc  amplitude:  average  amplitude  of  saccades  during  trials  in  which  anxiety  was 
reported. 

•  sacc  vel:  average  velocity  of  saccades  during  trials  in  which  anxiety  was  reported. 

•  pupil  size  variation:  maximum  pupil  size  change  over  a  4  second  window  relative  to 
average  pupil  size  over  the  past  ten  seconds.  By  using  the  running  average  of  pupil  size 
we  could  discount  the  effects  of  pupil  dilation  in  response  to  changing  light  levels  and 
other  extraneous  factors. 


The  graph  reveals  an  increase  in  all  five  measures  in  this  individual  as  self-reported  levels  of 
anxiety  increased.  The  pattern  was  repeated  across  the  two  other  subjects  who  also  reported 
reliable  increases  in  anxiety  levels. 


Muscular  contraction  associated  with  anxiety  was  measured  using  EMG.  As  per  the  below 
graph,  analysis  of  the  muscle  activation  patterns  of  the  same  participant  (subject  A)  revealed  an 
matching  increase  in  activity  associated  with  anxiety,  primarily  in  muscles  of  the  leg  with  the 
level  of  activation  increasing  by  up  to  50%  above  the  non- anxious  state.  Similar  patterns  were 
observed  in  the  other  subjects  reporting  anxiety,  who  also  showed  greater  amounts  of  muscle 
contraction  in  the  jaw,  shoulder  and  wrist  as  well  as  the  thigh  and  calf  muscles  in  the  higher 
anxiety  state. 


Subject  A:  Muscle  Activity  (overall  RMS) 

Activity  when  anxious  as  %  of  activity  when  not- 

anxious 


100%  ■ 


0%  ' — 


Jaw  Shoulder  Thigh  Calf 


7.  Discussion 


The  field  of  affective  computing  is  still  in  its  infancy.  The  purpose  of  the  experimental  set-up 
described  in  this  paper  was  to  cast  a  wide  net  across  a  range  of  physiological  measures  that  could 
be  reasonably  expected,  based  on  prior  research,  to  correlate  with  affective  state.  By  combining 
standard  measures  such  as  blink  rate  with  muscle  activation  and  other  eye  and  head  movement 
data,  our  aim  was  to  commence  the  development  of  an  objective  tool  able  to  draw  out  reliable 
predictors  of  affective  state  from  combinations  of  physiological  state  components.  As  an 
important  first  step,  this  paper  has  described  how  a  combination  of  commercially  available  eye¬ 
tracking  and  EMG  equipment  can  be  linked  to  provide  synchronised  data  acquisition  for  affect 
tracking  purposes. 


Although  the  project  was  unable  to  undertake  Pilot  Two  which  could  have  be  expected  to 
generate  higher  levels  of  stress  and  anxiety  as  the  simulator  exposure  would  have  been  in  a  real- 
world  training  scenario,  we  found  that  even  during  a  lower-anxiety  experimental  set-up 
simulation  the  expected  physiological  responses  to  increasing  anxiety  where  significant  enough 
to  be  clearly  captured  by  the  initial  pieces  of  objective  monitoring  equipment  trialled  in  Pilot 
One.  As  outlined  in  later  section  8  in  future  stages  of  the  project  additional  pieces  of  monitoring 
equipment  will  be  added  to  the  research  protocol  including  face  recognition  software  DeMoLib 
being  currently  developed  at  the  Australian  National  University,  EEG,  respiration  and  skin 
temperature. 


7.1  Implications  for  Affective  Skill  Theory 


Initial  findings  from  Pilot  One  indicate  that,  in  accordance  with  prior  literature  there  are 
relatively  robust  physiological  measures  which  can  be  monitored  during  simulation  training  to 
accurately  reflect  changes  in  affective  states  such  as  anxiety  and  stress.  The  development  of  an 
objective  measurement  tool  which  brings  together  each  of  these  physiological  measures  into  a 
real-time  affective  state  monitoring  device  will  provide  a  new  approach  to  objective 
measurement  of  preparedness  for  high  stress  operations. 


Overwhelmingly  to-date  affect  has  been  conceptualized  in  performance  theory  as  a  product  of 
how  difficult  or  dangerous  a  mission  is.  Yet  simulation-based  training  has  been  targeting  the 
strengthening  and  control  of  affective  skills  to  prepare  professionals  prior  to  undertaking 
missions  for  decades.  Simulators  have  always  been  popular  in  tenns  of  training  professionals  to 
‘keep  a  clear  head’  in  times  of  crisis.  The  very  nature  of  simulation  training  to  prepare  personnel 
makes  the  case  that  a  professional’s  affective  state  is  not  limited  to  being  a  by-product  of  his/her 
work  demands.  Simulation  has  focused  on  affect  because  it  is  acknowledged  that  intense  affect 
as  experienced  during  stress  can  negatively  affect  performance.  It  can  cause  overestimation  of 
risks  and  avoidant  choices  and  also  be  a  distraction  when  it  provides  infonnation  or  motivation 
to  attend  to  or  act  on  emotional  infonnation  at  the  expense  of  more  important  content  (Peters, 
Lipkus  &  Diefenbach,  2006).  In  elite  sports  a  professional  who  can  control  his  emotions  and 
purposefully  apply  them  to  achieve  optimal  performance  outcomes  is  described  as  being 
‘mentally  tough’  (Gordan  &  Gucciardi,  2010;  Gucciardi  &  Gordon,  2009). 


A  theoretical  framework  which  supports  the  study  of  affect  within  the  context  of  professional 
training  for  optimum  performance  under  high  stress  conditions  will  assist  in  the  investigation  of 
such  research  questions  as:  Can  a  level  of  “expert  affective  skill”  at  which  best  perfonnance 
outcomes  are  achieved  on  target  tasks  be  ascertained.  Can  the  ‘expert  affective  skill  level  be  used 
as  a  benchmark  against  which  a  novice’s  affective  performance  can  be  measured  to  ascertain 
whether  affective  levels  need  to  be  increased  or  decreased  to  support  best  perfonnance 
outcomes? 

The  design  of  simulation  training  provides  the  best  test  environment  for  AST.  Once  specific 
objectives  for  training  have  been  identified,  specialized  mission  scripts  and  trigger  events  are 
developed.  A  trigger  event  is  designed  to  elicit  a  specific  response  from  a  trainee,  instructional 
designers  work  with  subject  matter  experts  to  identify  critical  behaviors  that  should  unfold  after  a 
trigger  event.  .  The  instructor  knows  for  any  given  moment  in  a  scenario  what  objectives  are 
being  trained,  what  trigger  events  are  about  to  occur,  and  what  behaviors  are  critical  to  mission 
success  (Bennett,  Schreiber  &  Andrews,  2002). 

The  original  goal  of  the  AOARD  104011  project  was  to  explore  the  role  of  affect  in  objective 
training  evaluation.  Through  the  development  of  the  AST  theory  to  explain  and  support  the 
research  design  it  became  evident  that  AST  through  conceptualizing  affect  as  an  objective 
measureable  professional  skill  could  also  be  extended  to  provide  a  method  of  representing 


affective  workload  at  the  overall  mission  level  which  could  then  be  developed  into  a  predictive 
affective  workload  measure  to  support  operational  planning.  Predictions  of  operation  success  can 
be  modeled  based  on  the  ability  to  assess  the  match  between  the  overall  operational  affective 
requirements  with  the  individual  affective  skill  levels  of  personnel. 

In  practice  how  this  will  occur  is  measurements  of  an  individual’s  affective  skills,  taken  by  the 
objective  physiologically-based  tool  during  simulation  training,  will  be  integrated  with  a 
computational  perfonnance  prediction  model  which  plots  affect,  effort  and  task  demands  at  the 
overall  mission  level.  The  first  trial  of  the  Affective  Skills  Theory  in  the  area  of  operational 
planning  will  match  the  individual  affective  skill  level  of  professionals  undertaking  an  identified 
simulation  task  which  will  then  be  matched  by  affective  workload  predictions  made  for  that  same 
task  using  the  computational  version  of  the  compensatory  control  model  currently  under 
development.  This  application  is  discussed  further  in  the  following  section  7.3. 


7.3  The  role  of  Affective  Skill  Theory  in  Prediction/Operational  Planning. 

A  number  of  workload  prediction  models  have  been  developed  are  in  use.  Predictive  workload 
measures  use  models  that  represent  workload  for  the  purpose  of  prediction.  These  generally 
analyze  a  task/workload  based  on  multiple  resource  theory  of  human  attention,  resulting  in 
estimates  of  the  cognitive,  psychomotor,  and  sensory  components  of  workload  for  each  task.  The 
theory  underlying  TAWL  methodology  recognizes  five  independent  workload  components: 
auditory,  kinesthetic,  visual,  cognitive,  and  psychomotor  (Hamilton  &  Cross  1993).  The  ARL- 
HRED’s  Improved  Perfonnance  Research  Integration  Tool  (IMPRINT)  model  uses  the  visual, 
auditory,  cognitive  and  psychomotor  model  (VACP)  to  estimate  workload  based  on  scales 
developed  by  McCracken  and  Aldrich  (1984).  To  our  knowledge  no  current  prediction  models 
analyze  the  affective  cost  of  an  operation. 

Real-time  monitoring  of  individuals  for  affective  skill  capacity  via  an  approach  based  on 
sampling  multiple  neurophysiologic  signals,  and  integrating  those  signals  with  performance 
prediction  models  able  to  estimate  the  affective  component  of  workload,  potentially  provides  a 
method  of  supporting  commanders  and  control  personnel.  AST  identifies  a  new  measure  of 
performance  through  integrating  affective  skills  into  a  model  predicting  operational  performance 
this  will  enhance  the  quality  and  quantity  of  infonnation  supporting  decision  making  at  the 
operational  level. 

The  ability  to  control  affect  in  a  manner  that  uses  emotion  to  improve  performance  is  a  trainable 
skill.  Affective  skills  are  integral  to  success  as  affect  moderates  human  perfonnance  in  ways  that 
are  sometimes  difficult  to  predict.  The  positive  or  negative  effects  of  a  single  affective  event  may 
depend  on  operational  context,  individual  perception  or  degree.  For  example  a  small  amount  of 
fear  might  encourage  a  runner  to  run  faster,  but  a  large  amount  of  fear  might  make  him  unable  to 
run  at  all  (Redden  et  al,  2004). 

The  Affective  Skill  Theory  developed  in  the  first  phase  of  the  current  AOARD  project  supports 
the  future  development  of  a  predictive  workload  measure  able  to  estimate  affective  workload. 
AST  conceptualizes  affect  as  a  professional  skill  that  can  be  trained  prior  to  an  operation  and  can 


also  therefore  be  used  as  a  key  variable  in  evaluating  operation  success  based  on  the  ability  to 
assess  the  match  between  the  affective  resources  of  personnel  with  affective  requirements 
necessary  for  success  at  the  operational  level. 

The  development  of  a  theoretical  model  that  conceptualizes  affect  as  skill  necessary  to 
successfully  undertake  task  demands  has  significant  potential  to  assist  command  and  control  in 
the  area  of  operational  planning.  For  example,  a  model  of  task  workload  which  accounts  for 
affective  output  (the  affective  costs  of  a  deployment)  will  support  the  identification  of  situations 
in  the  field  which  will  impose  high  affective  task  demands  on  personnel.  When  an  operation  has 
been  identified  as  high  affective  demand,  the  optimal  affective  level  of  expertise  required  of 
personnel  to  successfully  complete  the  operation  can  be  estimated.  Ultimately  such  a  theoretical 
model  that  facilitates  the  mapping  of  affective  requirements  and  cost  with  task  demands  could 
assist  in  predictions  of  overall  mission  success  and  in  the  planning  of  additional  training 
requirements. 

Affective  state  is  a  key  variable  within  a  computational  prediction  model  currently  being 
developed.  The  predictive  computational  affect  workload  model  (CAWM)  will  ultimately  be 
able  to  offer  predictions  about  the  relationships  between  task  demands,  task  performance,  affect 
and  their  impact  on  goal  achievement.  The  innovation  of  the  CAWM  is  to  develop  a  quantitative 
model  which  translates  the  qualitative  compensatory  control  model  of  human  performance  under 
stress  and  high  workload  (Hockey,  1997)  into  a  computational  form.  The  compensatory  control 
model  proposes  that  physiological  and  affective  responses  are  inherent  in  the  regulatory  process. 
Previously  psychological  models  (such  as  the  compensatory  control  model)  have  been  based  on 
verbal  descriptions  of  the  relationships  between  variables.  These  models  allow  qualitative 
descriptions  about  human  performance,  but  do  not  allow  quantitative  assessments  to  be  made. 
Thus  the  initial  step  in  developing  a  quantitative  model  is  to  translate  the  existing  qualitative 
theories  into  a  computational  fonn.  The  initial  step  in  the  development  of  the  computational 
model  CAWM  is  forcing  the  parameters  of  the  compensatory  control  model  to  be  precisely 
identified  and  defined  to  allow  the  translation  of  the  existing  qualitative  theory  into  the 
computational  fonn  (Fletcher,  2010).  The  representation  of  relationships  between  task/overall 
mission  variables  as  mathematical  functions  provides  the  means  of  identifying  situations  where 
excessive  levels  of  task  demands  exist.  While  the  model  is  not  limited  to  the  variable  of  affect 
this  workload  variable  can  be  plotted  over  time. 

In  summary,  the  Affective  Skill  Theory  provides  a  new  approach  to  comparing  personnel 
affective  skills  levels  with  mission  affective  requirements.  The  current  AOARD  104011  project 
commenced  the  work  to  develop  an  objective  real-time  monitoring  tool  that  will  quantitatively 
assess  individual  personnel's  affective  levels  during  a  simulated  task.  The  CAWM  can  then  be 
used  to  plot  the  variables,  including  affect,  of  a  required  operation  over  time  and  detennine  the 
level  of  affective  skills  required  to  reach  goal  achievement.  If  the  model  highlights  a  mismatch 
between  personnel  affective  resources  and  affective  skills  necessary  for  mission  success  the 
framework  will  in  fact  pinpoint  type  and  degree  of  affective  training  requirements  pre¬ 
deployment  to  ensure  a  professional  affective  skill  and  mission  match. 


8.  Future  Research  Directions 


Pilot  One  commenced  with  EMG,  pupilometry  and  feature  extraction  methods  of  monitoring 
physiological  responses  to  affective  states.  In  this  experiment  anxiety,  as  measured  by  the 
MAACL-R,  was  the  affective  state  most  strongly  associated  with  the  experimental  simulation 
task.  Now  that  proof  of  concepts  and  physical  equipment  trialing  has  been  achieved  the  future 
phases  of  the  project  will  be  the  addition  and  trialing  of  further  physiological  monitoring  devices. 
The  larger  and  more  varied  the  final  data  set  the  greater  the  ability  of  the  objective  affect 
measurement  tool  (OAMT)  to  differentiate  between  individual  target  affective  states.  The  new 
devices  to  be  trialed  include: 


8.1  Face  Tracking  via  DeMoLib 


Facial  expressions  are  a  major  component  in  judging  person’s  affective  state  (Asthana,  Sargih, 
Wagner  &  Goecke,  2009).  Muscle  movements  involved  in  fear  include  raising  eyebrows  and 
drawing  them  together,  and/or  stretching  the  lips  horizontally  so  that  the  lips  form  a  rectangular 
mouth  shape.  In  more  extreme  states  of  fear,  a  raised,  tensed  upper  eyelid  which  widens  the  eye 
is  also  evident  (Harrigan  &  O’Connell). 


Dr  Roland  Goecke,  Head  of  Vision  and  Sensing  Group  at  the  University  of  Canberra,  Australia 
will  be  providing  his  developmental  software  DeMoLib  for  testing  with  the  OAMT  application. 
The  system  will  be  used  posthoc  initially  with  transition  into  real-time  monitoring  in  later  stages 


8.2  EEG 


New  brain  imaging  technologies  are  opening  the  windows  to  new  ways  of  looking  at  affective 
states  (AIZoubi,  Calvo,  Stevens,  2009).  Some  previous  studies  have  used  a  combination  of  EEG 
and  other  physiological  signals,  while  others  used  EEG  solely  for  affect  detection.  A  recent  study 
focussing  on  the  automatic  classification  techniques  that  could  be  used  for  EEG  data  showed  that 
accuracies  well  above  the  baseline  are  possible.  This  new,  exploratory  science  will  be 
investigated  for  correlations  with  the  other  measures  in  use  in  the  project. 


8.3  Bioharness 


While  feature  extraction  and  eye-tracking  are  the  focus  of  the  project,  the  end  goal  is  to  provide 
multiple  measures  to  yield  independent  and  converging  evidence  of  affective  states.  For  that 
reason,  the  study  will  also  include  several  physiological  measures  (in  addition  to  EMG). 
Physiologists  demonstrate  that  high  stress  level  is  accompanied  with  the  symptoms  of  faster 
heart  beat,  rapid  breathing,  increased  sweating,  cool  skin,  feelings  of  nausea,  tense  muscles  and 
alike.  (Liao,  Zhang,  Zhu  &  Ji,  2005).  We  have  purchased  a  Zephyr  BioHamess  for  use  in  this 
project.  It  uses  wireless  technology  to  provide  real-time  data  on  heart  rate,  respiration  and  skin 
temperature.  It  will  be  trialed  for  the  first  time  when  Pilot  Study  Two  is  undertaken. 


8.4  Linking  Affect  to  Perceptual  Processing 


If  one  wishes  to  synchronise  external  scene  events  with  emotional  response,  a  record  of  those 
events  is  required.  The  eye  tracking  system  we  are  using  has  the  advantage  that  it  can  also  record 
the  scene  observed  by  the  participant  via  a  head-mounted  camera.  Eye  tracking  data  is  rendered 
onto  the  camera’s  video  stream  as  a  small  fixation  cross  and  this  video  is  automatically 
synchronised  with  the  eye  movement  data.  This  allows  offline  matching  of  salient  visual  events 
(e.g.  obstacle  appears,  target  acquired)  with  the  physiological  data. 


People  can  learn  to  distinguish  between  task-relevant  information  and  task-irrelevant  information 
(Sohn,  Douglass,  Chen  &  Anderson,  2000).  Through  eye-tracking  data,  learning  should  be 
reflected  in  the  pattern  of  attention  distribution  or  eye  fixation.  In  simulation  training  we  want  to 
know  what  is  happening  with  participant’s  perceptual  processes  at  specific  decision  points  within 
the  scene  and  the  subsequent  impact  of  evoked  emotion.  In  successful  training  participants 
should  leam  to  pay  more  attention  to  on-task  regions  relative  to  off-task  regions  (Sohn  et  al, 
2000).  If  the  training  is  properly  designed,  eye-movement  data  should  show  that  users  look  at 
irrelevant  regions  less  and  less  as  they  practice  more. 


We  propose  that  by  monitoring  affective  state  while  simultaneously  tracking  people’s  perceptual 
processing  was  can  ascertain  the  impact  of  stress  at  different  decision  points.  It  is  expected  that 
high  affect  levels  of  negative  may  negatively  influence  visual  processing  of  cues.  For  example 
when  angry,  a  person  may  focus  for  less  time  on  an  important  cue  they  had  spent  far  longer 
visually  processing  during  an  earlier  training  session.  The  ability  to  synthetically  evoke  intense 


affective  states  during  a  complex  cognitive  task  allows  accurately  timed  measurement  of  user 
responses  and  will  help  to  answer  such  questions  as  is  there  an  optimal  degree  of  affective 
intensity  which  supports  the  laying  down  of  a  long  term  memory  trace  but  over  which  intensity 
level  cognition  degrades?  Such  insights  will  assist  the  development  of  a  model  to  guide  use  of 
eye  movement  monitoring  to  evaluate  high  affect  training  in  virtual  environments. 


9.  References 

Ahn  SJ,  Bailenson  J,  Fox  J,  Jabon  M  (2009)  Using  Automated  Facial  Expression  Analysis  for  Emotion 
and  Behavior  Prediction.  Paper  presented  at  the  annual  meeting  of  the  NCA  95th  Annual  Convention, 
Chicago,  II,  Nov  11,  2009  Accessed  Online  24/8/2010  at  http://vhil.stanford.edu/pubs/20 1 0/ahn-hemm- 
facial-expression.pdf 

Asthana  A,  Saragih  J,  Wagner  M  ,  Goecke  R(2009)  Evaluating  AAM  Fitting  Methods  for  Facial 
Expression  Recognition.  Proceedings  of  the  IEEE  International  Conference  on  Affective  Computing  and 
Intelligent  Interaction  ACII2009,  pp598-605,  Amsterdam,  Netherlands,  10-12  Sep.  2009.  IEEE. 

AIZoubi  O,  Calvo  R  A,  Stevens  RH  (2009)  Classification  of  EEG  for  Affect  recognition:  An  adaptive 
approach,  AI  2009:Advances  in  Artificial  Intelligence,  Proceedings  (LNAI)  Vol  5866  pp52-61 

Bailenson  JN,  Pontikakis  ED,  Mauss  IB,  Gross  JJ,  Jabon  ME,  Hutcherson  CAC,  Nass  C,  Oliver  J  (2008) 
Real-time  classification  of  evoked  emotions  using  facial  feature  tracking  and  physiological  responses, 
International  Journal  of  Human-Computer  Studies,  66,  pp. 303-3 17 

Ballard  TJ  (2009)  The  role  of  affect  in  goal  setting  and  goal  striving:  A  multilevel  analysis,  Honors 
Thesis  in  Psychological  Science  at  the  University  of  Queensland. 

Bennett  W  Jr,  Schreiber  BT,  Andrews  D  (2002)  Developing  competency-based  methods  for  near-real- 
time  air  combat  problem  solving  assessment.  Computers  in  Human  Behavior,  18  pp773-782. 

Blewett  WK,  Ramos  GA,  Redmond  DP,  Cadarette  BS,  Hugens  GA,  Fatkin  LT,  McKiernan  K  (1994) 
p2NBC2  test:  The  effects  of  microclimate  cooling  on  tactical  performance  (Technical  Report,  148) 
Aberdeen  Proving  Ground,  MD:  US  Amy  Chemical  and  Biological  Defense  Agency. 

Chapman  C  ,  Oka  S,  Bradshaw  DH,  Jacobson  RC,  Donaldson,  GW  (1999)  Phasic  pupil  dilation  response 
to  noxious  stimulation  in  normal  volunteers:  Relationship  to  Brain  evoked  potentials  and  pain  report, 
Psychophysiology,  36(1),  pp44-52. 

Chatterton  RT  Jr.,  Vogelsong  KM,  Lu  Y,  Ellman  AB,  Hudgens  GA  (1996)  Salivary  amylase  as  a  measure 
of  endogenous  adrenergic  activity.  Clinical  Psychology,  16  pp433-438. 

Chi  MTH,  Glaser  R,  Farr  MJ  (Eds)  (1988)  The  nature  of  expertise.  Hillsdale,  NJ:Erlbaum. 

Connaughton  D,  Wadey  R,  Hanton  S,  Jones  G  (2008)The  development  and  maintenance  of  mental 
toughness:  Perceptions  of  elite  performers.  Journal  of  Sports  Sciences,  26,  pp83-95. 

Damasio,  A.R.  (1995).  L’erreur  de  Descartes.  La  raison  des  emotions.  Paris  :  Eds  Odile  Jacob  (Sciences). 


Daniels  LM,  Stupnisky  RH,  Pekrun  R,  Haynes  TL,  Perry  RP,  Newall  NE  (2009)  A  longitudinal  analysis 
of  achievement  goals:  From  Affective  antecedents  to  emotional  effects  and  achievement  outcomes, 
Journal  of  Educational  Psychology,  101(4)  pp948-963. 

El  Kaliouby  R,  Robinson  P  (2005)  Real-time  inference  of  complex  mental  states  from  facial  expressions 
and  head  gestures.  In:  Real-time  Vision  for  HCI.  Springer,  pp.  181-200.  ISBN  0-387-27697-1. 

Elliot  AJ,  Thrash  TM  (2002)  Approach-avoidance  motivation  in  personality:  Approach  and  avoidance 
temperaments  and  goals.  Journal  of  Personality  and  Social  Psychology,  82  pp804-818. 

Ericsson  KA,  Chamess  N,  Feltovich  PJ,  Hoffman  RR  (2006)  The  Cambridge  handbook  of  expertise  and 
expert  performance.  Cambridge,  England:  Cambridge  University  Press. 

Fletcher  K  (2010)  Towards  a  Computational  Compensatory  Control  Model  of  Mental  Workload. 
Unpublished  manuscript,  The  University  of  Queensland. 

Foo  M,  Uy  MA,  Baron  RA  (2009)  How  do  feelings  influence  effort?  An  empirical  study  of  entrepreneurs’ 
affect  and  venture  effort.  Journal  of  Applied  Psychology,  94(4)  pp  1086- 1094. 

Gee  P  (2010)  Measuring  emotions  throughout  skill  acquisition:  A  multi-level  analysis,  Proceedings  of 
ICAP,  Melbourne  11-16  July,  2010. 

Gordon  S,  Gucciardi  D  (2010)  Mental  Toughness.  In  J.  Adams  (Ed.),  Sport  psychology:  Theory  and 
practice.  Pearson  (In  Press). 

Gray  JA  (1990)  Brain  systems  that  mediate  both  emotion  and  cognition.  Cognition  &  Emotion.  Special 
Issue:  Development  of  relationships  between  emotions  and  cognition,  4(3)  pp269-288. 

Gucciardi  D,  Gordon  S  (2009)  Development  and  preliminary  validation  of  the  Cricket  Mental  Toughness 
Inventory  (CMTI),  Journal  of  Sport  Sciences,  27(  12)pp  1293- 13 1 0. 

Hamilton  DB,  Cross  KD  (1993)  Preliminary  validation  of  the  task  analysis/workload  methodology.  Fort 
Rucker,  AL:  ANACAPA  Sciences. 

Harrigan  JA,  O’Connell  DM  (1996)  How  do  you  look  when  feeling  anxious?  Facial  displays  of  anxiety. 
Personality  and  Individual  Differences  Vol  21,  2,  pp205-212 

Hockey  GRJ  (2005)  Operator  functional  state:  The  prediction  of  breakdown  in  human  performance.  In  J 
Duncan,  L  Phillips  &  P  McLeod  (Eds)  Measuring  the  Mind:  Speed,  Control  and  Age  (pp373-394). 

Oxford:  Oxford  University  Press. 

Hockey  GRJ  (1997)  Compensatory  control  in  the  regulation  of  human  performance  under  stress  and  high 
workload:  A  cognitive-energetical  framework.  Biological  Psychology,  45  pp73-93. 

Koestner  R,  Lekes  N,  Powers  TA,  Chicoine  E  (2002)  Attaining  personal  goals:  Self-concordance  plus 
implementation  intentions  equals  success.  Journal  of  Personality  and  Social  Psychology,  83(1)  pp231- 
244. 


Lazarus,  R.S.  (1966).  Psychological  Stress  and  the  Coping  Process.  New  York:  McGraw-Hill. 


Lehrer  J  (2009)  ‘Deliberate  calm’  guided  US  airways  crew,  The  Los  Angeles  Times. 


Liao  W,  Zhang  W,  Zhu  Z,  Ji  Q,  Gray  WD  (2006)  Toward  a  decision-theoretic  framework  for  affect 
recognition  and  user  assistance.  International  Journal  of  Human-Computer  Studies,  64(9),  pp. 847-873. 

Linnenbrink  EA,  Pintrich  PR  (2002)  Achievement  goal  theory  and  affect:  An  symmetrical  bidirectional 
model.  Educational  Psychologist,  37,  pp69-78. 

Loft  S,  Bolland  S,  Humphreys  M  ,  Neal  A  (2009)  A  Theory  and  Model  of  Conflict  Detection  in  Air 
Traffic  Control:  Incorporating  Environmental  Constraints,  Journal  of  Experimental  Psychology:  Applied, 
15(2)  pp  106- 124. 

Louro  MJ,  Pieters  R,  Zeelenberg  M  (2007)  Dynamics  of  multiple-goal  pursuit.  Journal  of  Personality  and 
Social  Psychology,  93(2)  ppl74-193. 

Matsumoto  D,  Wilson  J  (2008).  Culture,  emotion,  and  motivation.  In  R.  Sorrentino  and  S.  Yamaguchi 
(eds.),  Handbook  of  Motivation  and  Cognition  across  Cultures  (pp.  541-563).  New  York:  Elsevier. 

Mavin  TJ.  (2010)  Assessing  pilot’s  performance  for  promotion  to  airline  captain.  Thesis  for  the  degree  of 
Doctor  of  Education,  The  University  of  Queensland. 

McCracken  JH,  Aldrich  TB  (1984)  Analyses  of  selected  LHX  mission  functions:  Implications  for 
operator  workload  and  system  automation  goals  (Technical  Note  AS1479-024-84).  Fort  Rucker,  AL:  US 
Army  Research  Institute  Aviation  Research  and  Development  Activity. 

Partla  T,  Surakka  V  (2003)  Pupil  size  variation  as  an  indication  of  affective  processing.  International 
Journal  of  Human-Computer  Studies,  59,  pp  185- 198 

Pekrun  R,  Elliot  AJ,  Maier  MA  (2006)  Achievement  goals  and  discrete  emotions:  A  theoretical  model 
and  prospective  test.  Journal  of  Educational  Psychology,  98  pp583-597. 

Redden  ES,  Sheehy  JB,  Bjorkman  EA  (2004)  The  study  and  measurement  of  human  performance  by 
military  service  laboratories.  Advances  in  Human  Performance  and  Cognitive  Engineering  Research,  5, 
pp5 17-59. 

Reynolds  C,  Picard  RW  (2005)  Evaluation  of  affective  computing  systems  from  a  dimensional 
metaethical  position.  1st  Augmented  Cognition  Conference,  in  conjunction  with  the  1 1th  International 
Conference  on  Human-Computer  Interaction,  July  22-27,  2005  Las  Vegas,  NV. 

Russo  MB,  Stetz  MC,  Thomas  ML  (2005)  Monitoring  and  predicting  cognitive  state  and  performance  via 
physiological  correlates  of  neuronal  signals,  Aviation,  Space  and  Environmental  Medicine,  76,  7,  ppC-59- 
C63. 

Seo  M,  Barrett  LF,  Bartunek  JM  (2004)  The  role  of  affective  experience  in  work  motivation.  Academy  of 
Management  Review,  29(3)  pp423-439. 

Sohn  M-H,  Douglass  S  A,  Chen  M-C,  Anderson  J  R  (2000).  Eye-movements  during  unit-task  execution 
in  a  complex  problem-solving  situation.  In  Proceedings  of  the  44th  Annual  Meeting  of  the  Human  Factors 
and  Ergonomics  Society,  pp378-381 


Susskind  JM,  Lee  DH,  Cusi  A  ,Feiman  R,  Grabski  W,  Anderson  AK  (2008)  Expressing  fear  enhances 
sensory  acquisition.  Nature  Neuroscience,  1 1(7),  pp843-850. 

Tichon,  J,  (2010)  Evaluation  of  Virtual  Reality  Training  using  Affect,  International  Journal  on  E- 
Leaming.  Under  Review. 

Vealey  RS  (2007).  Mental  skills  traning  in  sport.  In  G  Tenenbaum  &  RC  Eklund  (Eds)  Elandbook  of  sport 
psychology  (pp. 287-309).  Eloboken,  NJ:  Wiley. 

Yoshie  M,  Kudo  K,  Ohtsuki  T  (2008)  Effects  of  psychological  stress  on  state  anxiety,  electromyographic 
activity,  and  arpeggio  performance  in  pianists.  Medical  Problems  of  Performing  Artists.  (Sept.)  pp.120- 
132. 


